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Abstract 

Algorithmic decision making systems are 
ubiquitous across a wide variety of online as 
well as offline services. These systems rely on 
complex learning methods and vast amounts 
of data to optimize the service functionality, 
satisfaction of the end user and profitabil¬ 
ity. However, there is a growing concern that 
these automated decisions can lead, even in 
the absence of intent, to a lack of fairness, 
i.e., their outcomes can disproportionately 
hurt (or, benefit) particular groups of peo¬ 
ple sharing one or more sensitive attributes 
(e.g., race, sex). In this paper, we introduce 
a flexible mechanism to design fair classifiers 
by leveraging a novel intuitive measure of de¬ 
cision boundary (un)fairness. We instantiate 
this mechanism with two well-known classi¬ 
fiers, logistic regression and support vector 
machines, and show on real-world data that 
our mechanism allows for a fine-grained con¬ 
trol on the degree of fairness, often at a small 
cost in terms of accuracy. 

A Python implementation of our mechanism 
is available at fate-computing.mpi-sws.org 


1 INTRODUCTION 

Algorithmic decision making processes are increasingly 
becoming automated and data-driven in both online 
(e.g., spam filtering, product personalization) as well 
as offline (e.g., pretrial risk assessment, mortgage ap¬ 
provals) settings. However, as automated data analy¬ 
sis replaces human supervision in decision making, and 
the scale of the analyzed data becomes “big”, there are 


growing concerns from civil organizations Bhandari 
governments 


2016 


2016 


and researchers Sweeney 2013 about potential 


Podesta et al. 2014 Munoz et al. 


loss of transparency, accountability and fairness. 


Anti-discrimination laws in many countries prohibit 


Proceedings of the 20 th International Conference on Artifi¬ 
cial Intelligence and Statistics (AISTATS) 2017, Fort Laud¬ 
erdale, Florida, USA. JMLR: W&CP volume 54. Copy¬ 
right 2017 by the author(s). 


unfair treatment of people based on certain attributes, 
also called sensitive attributes, such as gender or 

These laws typically 


race 


Civil Rights Act 1964 


evaluate the fairness of a decision making process by 


means of two distinct notions Barocas and Selbst 


2016 : disparate treatment and disparate impact. 


A decision making process suffers from disparate treat¬ 
ment if its decisions are (partly) based on the sub¬ 
ject’s sensitive attribute information, and it has dis¬ 
parate impact if its outcomes disproportionately hurt 
(or, benefit) people with certain sensitive attribute va¬ 
lues (e.g., females, blacks). 


While it is desirable to design decision making sys¬ 
tems free of disparate treatment as well as disparate 
impact, controlling for both forms of unfairness simul¬ 
taneously is challenging. One could avoid disparate 
treatment by ensuring that the decision making pro¬ 
cess does not have access to sensitive attribute infor¬ 
mation (and hence cannot make use of it). However, 
ignoring the sensitive attribute information may still 
lead to disparate impact in outcomes: since automated 
decision-making systems are often trained on histori¬ 
cal data, if a group with a certain sensitive attribute 
value was unfairly treated in the past, 0 this unfair¬ 
ness may persist in future predictions through indi¬ 
rect discrimination Pedreschi et al. |2008| , leading to 
disparate impact. Similarly, avoiding disparate im¬ 
pact in outcomes by using sensitive attribute informa¬ 
tion while making decisions would constitute disparate 
treatment, and may also lead to reverse discrimina¬ 
tion 


Ricci vs. DeStefano 2009 


In this work, our goal is to design classifiers— 
specifically, convex margin-based classifiers like logistic 
regression and support vector machines (SVMs)—that 
avoid both disparate treatment and disparate impact, 
and can additionally accommodate the “business ne¬ 
cessity” clause of disparate impact doctrine. Accord- 


Tike earlier studies on fairness aware-learning, we as¬ 
sume that while historical class labels may be biased 
against group(s) with certain sensitive attribute value(s), 
they still contain some degree of information on the true 
(unbiased) labels. Assuming the class labels to be com¬ 
pletely biased and having no information on the true labels 
would render a learning task on such a dataset pointless. 
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ing to the business necessity clause, an employer can 
justify a certain degree of disparate impact in order to 


meet certain performance-related constraints Barocas 


and Selbst 2016 . However, the employer needs to en¬ 


sure that the current decision making incurs the least 
possible disparate impact under the given constraints. 
To the best of our knowledge, this clause has not been 
addressed by any prior study on fairness-aware lear¬ 
ning. 

While there is no specific numerical formula laid out by 
anti-discrimination laws to quantify disparate impact, 
here we leverage a specific instantiation supported 
by the U.S. Equal Employment Opportunity Com¬ 
mission: the “80%-rule” (or more generally, the p%- 
rule) [Biddle 2005 . The p%-rule states that the ratio 
between the percentage of subjects having a certain 
sensitive attribute value assigned the positive decision 
outcome and the percentage of subjects not having 
that value also assigned the positive outcome should 
be no less than p:100. Since it is very challenging 
to directly incorporate this rule in the formulation of 
convex margin-based classifiers, we introduce a novel 
intuitive measure of decision boundary (un)fairness as 
a tractable proxy to the rule: the covariance between 
the sensitive attributes and the (signed) distance be¬ 
tween the subjects’ feature vectors and the decision 
boundary of the classifier. 


Our measure of fairness allows us to derive two com¬ 
plementary formulations for training fair classifiers: 
one that maximizes accuracy subject to fairness con¬ 
straints, and enables compliance with disparate impact 
doctrine in its basic form ( i.e ., the p%-rule); and an¬ 
other that maximizes fairness subject to accuracy con¬ 
straints, and ensures fulfilling the business necessity 
clause of disparate impact. Remarkably, both formu¬ 
lations also avoid disparate treatment, since they do 
not make use of sensitive attribute information while 
making decisions. Our measure additionally satisfies 
several desirable properties: (i) for a wide variety of 
convex margin-based (linear and non-linear) classifiers, 
it is convex and can be readily incorporated in their 
formulation without increasing their complexity; (ii) 
it allows for clear mechanism to trade-off fairness and 
accuracy; and, (iii) it can be used to ensure fairness 
with respect to several sensitive attributes. Experi¬ 
ments with two well-known classifiers, logistic regres¬ 
sion and support vector machines, using both synthetic 
and real-world data show that our fairness measure al¬ 
lows for a fine-grained control of the level of fairness, 
often at a small cost in terms of accuracy, and provides 
more flexibility than the state-of-the-art (see Table [T]) . 

Related Work. A number of prior studies have fo¬ 
cused on controlling disparate impact and/or disparate 
treatment-based discrimination in the context of bi¬ 


nary classification Romei and Ruggieri 2014 . These 
studies have typically adopted one of the two following 
strategies: 


The first strategy consists of pre-processing the train- 


ing data Dwork et al. 

2012 

Feldman et al. 

2015 

Kamiran and Calders 2 

009 

2 

010 . This typically in- 


volves (i) changing the value of the sensitive attributes 
or class labels of individual items in the training data, 
or (ii) mapping the training data to a transformed 
space where the dependencies between sensitive attri¬ 
butes and class labels disappear. However, these ap¬ 
proaches treat the learning algorithm as a black box 
and, as a consequence, the pre-processing can lead to 
unpredictable losses in accuracy. 


The second strategy consists of modifying exist¬ 


ing classifiers to limit discrimination Calders and 


Verwer[ |2010[ |Kamishima et ah| |2011 Goh et al. 


2016 . Among them, the work by Kamishima et al. 


Kamishima et al. 2011 is the most closely related to 


ours: it introduces a regularization term to penalize 
discrimination in the formulation of the logistic regres¬ 
sion classifier. 


Recently, Zemel et al. 2013 , building on Dwork et al. 


2012 , combined both strategies by jointly learning a 


fair representation of the data and the classifier para¬ 
meters. This approach has two main limitations: i) it 
leads to a non-convex optimization problem and does 
not guarantee optimality, and ii) the accuracy of the 
classifier depends on the dimension of the fair repre¬ 
sentation, which needs to be chosen rather arbitrarily. 


Many of the prior studies suffer from one or more of 
the following limitations: (i) they are restricted to a 
narrow range of classifiers, (ii) they only accommodate 
a single, binary sensitive attribute, and (iii) they can¬ 
not eliminate disparate treatment and disparate im¬ 
pact simultaneously. Table [l] compares the capabilities 
of different methods while achieving fairness. 


Finally, as discussed earlier, disparate impact is partic¬ 
ularly well suited as a fairness criterion when histori¬ 
cal decisions used during the training phase are biased 
against certain social groups. In such contexts, propor¬ 
tionality in outcomes ( e.g ., p%-rule) may help mitigate 
these historical biases. However, in cases where the 
(unbiased) ground-truth is available for the training 
phase, i.e., one can tell whether a historical decision 
was right or wrong , disproportionality in outcomes can 
be explained by the means of the ground-truth. In 
those cases, disparate impact may be a rather mis¬ 
leading notion of fairness, and other recently proposed 
criteria like “disparate mistreatment” by [Zafar et al. 


2017 , may be better suited notions of fairness. For 


more discussion into this alternative notion, we point 


the reader to our companion paper Zafar et al. 2017 




































































Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, Krishna P. Gummadi 


Method 

Disp. 

Treat. 

Disp. 

Imp. 

Business. 

Necessity 

Polyvalent 
sens, attrs. 

Multiple 
sens, attrs. 

Range of 
classifiers 

Our method 

/ 

y 

/ 

/ 

/ 

Any convex margin-based 

Kamiran and Calders (2010 

/ 

y 

X 

X 

X 

Any score-based 

Calders and Verwer 2010| 

/ 

y 

X 

X 

X 

Naive Bayes 

Luong et al. 2011 


/ 

X 

X 

X 

X 

Any score-based 

Kamishima et al. | 

2011] 

X 

y 

X 

X 

X 

Logistic Regression 

Zemel et al. 2013 


/ 

y 

X 

X 

X 

Log loss 

Feldman et al. 2015] 

/ 

y 

X 

y 

/ 

Any (only numerical non-sens, attrs.) 

Goh et al. 2016] 

/ 

y 

X 

y 

/ 

Ramp loss 


Table 1: Capabilities of different methods in eliminating disparate impact and/or disparate treatment. None of 
the prior methods addresses disparate impact’s business necessity clause. Many of the methods do not generalize 
to multiple (e.g., gender and race) or polyvalent sensitive attributes ( e.g ., race, that has more than two values). 


2 FAIRNESS IN CLASSIFICATION 

For simplicity, we consider binary classification tasks 
in this work. However, our ideas can be easily ex¬ 
tended to m-ary classification. 

In a binary classification task, one needs to find a 
mapping function /(x) between user feature vectors 
x € and class labels y £ {—1,1}. This task is 
achieved by utilizing a training set, {(xj, yi)}f = \, t° 
construct a mapping that works well on an unseen 
test set. For margin-based classifiers, finding this map¬ 
ping usually reduces to building a decision boundary 
in feature space that separates users in the training set 
according to their class labels. One typically looks for 
a decision boundary, defined by a set of parameters 0*, 
that achieves the greatest classification accuracy in a 
test set, by minimizing a loss function over a training 
set L(0), i.e., 0* = argmin e L(0). Then, given an un¬ 
seen feature vector Xj from the test set, the classifier 
predicts / e (x.j) = 1 if de*(xj) > 0 and fe(*i) = -1 
otherwise, where de*(x) denotes the signed distance 
from the feature vector x to the decision boundary. 


If class labels in the training set are correlated with 
one or more sensitive attributes {z i}fL x (e.g., gender, 
race), the percentage of users with a certain sensitive 
attribute having c?e*(xi) > 0 may differ dramatically 
from the percentage of users without this sensitive at¬ 
tribute value having de*(xj) > 0 ( i.e ., the classifier 
may suffer from disparate impact). Note that this may 
happen even if sensitive attributes are not used to con¬ 
struct the decision boundary but are correlated with 
one or more of user features, through indirect discrim¬ 
ination 


Pedreschi et al. 2008 


2.1 Fairness Definition 

First, to comply with disparate treatment criterion 
we specify that sensitive attributes are not used in de¬ 
cision making, i.e., {xj}^ and {z i }fL 1 consist of dis¬ 
joint feature sets. 


(or more generally the “p%-rule”), if the ratio between 
the percentage of users with a particular sensitive at¬ 
tribute value having dg(x) > 0 and the percentage of 
users without that value having dg(x) > 0 is no less 
than 80:100 (p:100). For a given binary sensitive at¬ 
tribute z £ {0,1}, one can write the p%-rule as: 


( P(d e {x )>0 


\P(de(x)>0 


P(dg (x)>0 

2=0) 

P(dg (x)>0 

2=1) 


> 


( 1 ) 


Unfortunately, it is very challenging to directly in¬ 
corporate the p%-rule in the formulation of convex 
margin-based classifiers, since it is a non-convex func¬ 
tion of the classifier parameters 0 and, therefore, it 
would lead to non-convex formulations, which are dif¬ 
ficult to solve efficiently. Secondly, as long as the user 
feature vectors lie on the same side of the decision 
boundary, the p%-rule is invariant to changes in the 
decision boundary. In other words, the p%-rule is a 
function having saddle points. The presence of saddle 
points furthers complicate the procedure for solving 
non-convex optimization problems Dauphin et al., 


2014 . To overcome these challenges, we next intro¬ 


duce a novel measure of decision boundary (un) fairness 
which can be used as a proxy to efficiently design clas¬ 
sifiers satisfying a given p%-rule. 


3 OUR APPROACH 

In this section, we first introduce our measure of de¬ 
cision boundary (un)fairness, the decision boundary 
covariance. We then derive two complementary for¬ 
mulations. The first formulation ensures compliance 
with disparate impact doctrine in its basic form (en¬ 
sure a given p%-rule) by maximizing accuracy sub¬ 
ject to fairness constraints. The second formulation 
guarantees fulfilling disparate impact’s “business ne¬ 
cessity” clause by maximizing fairness subject to ac¬ 
curacy constraints. 


Next, as discussed in Section [l] our definition of 


disparate impact leverages the “80%-rule” Biddle 


2005 . A decision boundary satisfies the “80%-rule” 


For conciseness, we append a constant 1 to all fea¬ 
ture vectors (xi) so that the linear classifier decision 
boundary equation 0 1 x + b = 0 reduces to 0 T x = 0. 
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3.1 Decision Boundary Covariance 

Our measure of decision boundary (un)fairness is de¬ 
fined as the covariance between the users’ sensitive 
attributes, {zand the signed distance from 
the users’ feature vectors to the decision boundary, 
i-e.: 

Cov(z ,dg(x)) = E[(z - z)de(x)]-E[(z - z)]d 0 (x) 

1 N 

~ -z)d 0 (x i ), (2) 

where E[(z — z)]dg(x) cancels out since E[(z — z)] = 0. 
Since in linear models for classification, such as logistic 
regression or linear SVMs, the decision boundary is 
simply the hyperplane defined by 9 r x = 0, Eq. ©> 
reduces to Y, =i ( z « “ z ) # T Xi. 

In contrast with the p%-rule (Eq. [l]), the decision 
boundary covariance (Eq. [2]) is a convex function with 
respect to the decision boundary parameters 9 , since 
de(x,;) is convex with respect to 6 for all linear, con¬ 
vex margin-based classifiers. [^] Hence, it can be easily 
included in the formulation of these classifiers without 
increasing the complexity of their training. 

Moreover, note that, if a decision boundary satisfies 
the “100%-rule”, i.e., 

P(de{x) > 0\z = 0) = P(d g (x) > 0\z = 1), (3) 

then the (empirical) covariance will be approximately 
zero for a sufficiently large training set. 

3.2 Maximizing Accuracy Under Fairness 
Constraints 

In this section, we design classifiers that maximize 
accuracy subject to fairness constraints ( e.g., a specific 
p%-rule), and thus may be used to ensure compliance 
with the disparate impact doctrine in its basic form. 

To this end, we find the decision boundary parameters 
6 by minimizing the corresponding loss function over 
the training set under fairness constraints, i.e.: 

minimize L{9) 

subject to ^ XI ill ( z i - z ) defa) < c, (4) 

^ E^Ii ( z i - z ) > -c, 

where c is the covariance threshold, which specifies an 
upper bound on the covariance between each sensitive 
attribute and the signed distance from the feature vec¬ 
tors to the decision boundary. In this formulation, c 
trades off fairness and accuracy, such that as we de¬ 
crease c towards zero, the resulting classifier will sat¬ 
isfy a larger p%-rule but will potentially suffer from a 

2 For non-linear convex margin-based classifiers like non¬ 
linear SVM, equivalent of de(x;) is still convex in the trans¬ 
formed kernel space. See Appendix [A] for details. 


larger loss in accuracy. Note that since the above op¬ 
timization problem is convex, our scheme ensures that 
the trade-off between the classifier loss function and 
decision boundary covariance is Pareto optimal. 

Remarks. It is important to note that the distance to 
the margin, dg(x), only depends on the non-sensitive 
features x and, therefore, the sensitive features z are 
not needed while making decisions. In other words, we 
account for disparate treatment , by removing the sensi¬ 
tive features from the decision making process and, for 
disparate impact , by adding fairness constraints dur¬ 
ing the training process of the classifier. Addition¬ 
ally, the constrained optimization problem Q, can 
also be written as a regularized optimization problem 
by making use of its dual form, in which the fairness 
constraints are moved to the objective and the corre¬ 
sponding Lagrange multipliers act as regularizers. 

Next, we specialize problem Q for logistic regression 
classifiers. 


Logistic Regression. In logistic regression classi¬ 
fiers, one maps the feature vectors x* to the class labels 
yt by means of a probability distribution: 

P{Vi = l| x i- O) = 1 e -dT Xi > ( 5 ) 


where 0 is obtained by solving a maximum like¬ 
lihood problem over the training set, i.e., 9* = 
argming — YiLi l°gp(2/il x i, 9). Thus, the correspond¬ 
ing loss function is given by — YiLi l°g|x^, 9), and 
problem Q adopts the form: 


minimize — 
subject to 


Eili logp(2/i|xi,0) 

(z* - z) 9 T *i < c, 
£££i (z< - z) 9 T Xi > -c, 


( 6 ) 


Appendix [A| presents the specialization of our formu¬ 
lation for both linear and non-linear SVM classifiers. 


3.3 Maximizing Fairness Under Accuracy 
Constraints 

In the previous section, we designed classifiers that 
maximize accuracy subject to fairness constraints. 
However, if the underlying correlation between the 
class labels and the sensitive attributes in the train¬ 
ing set is very high, enforcing fairness constraints 
may results in underwhelming performance (accuracy) 
and thus be unacceptable in terms of business objec¬ 
tives. Disparate impact’s “business necessity” clause 
accounts for such scenarios by allowing some degree 
of disparate impact in order to meet performance con¬ 
straints. However, the employer needs to ensure that 
the decision making causes least possible disparate 
impact under the given performance (accuracy) con- 

To accommodate 


straints 


Barocas and Selbst 2016 







Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, Krishna P. Gummadi 


such scenarios, we now propose an alternative formula¬ 
tion that maximizes fairness (minimizes disparate im¬ 
pact) subject to accuracy constraints. 

To this aim, we find the decision boundary parameters 
6 by minimizing the corresponding (absolute) decision 
boundary covariance over the training set under con¬ 
straints on the classifier loss function, i.e.: 


and also provides guarantees on avoiding negative clas¬ 
sification of certain individual users or group of users. 


Here, we adopt the p%-rule Biddle 2005 


as our 


true measure of fairness. However, as shown in Ap¬ 
pendix m we obtain similar results if we consider 
another measure of fairness used by some of the pre¬ 
vious studies in this area. 


minimize I ]? ( z i ~ z ) d e ( x i) 
subject to L{6) < (1 + 7 )L(B*), 


(7) 


where L(9*) denotes the optimal loss over the training 
set provided by the unconstrained classifier and 7 > 0 
specifies the maximum additional loss with respect to 
the loss provided by the unconstrained classifier. Here, 
we can ensure maximum fairness with no loss in accu¬ 


racy by setting 7 = 0. As in Section |3i2| it is possible 
to specialize problem (J7| for the same classifiers and 
show that the formulation remains convex. 


Fine-Grained Accuracy Constraints. In many 
classifiers, including logistic regression and SVMs, the 
loss function (or the dual of the loss function) is addi¬ 
tive over the points in the training set, i.e., L(6) = 
Si =1 £*(#), where Li{6) is the individual loss associ¬ 
ated with the i-th point in the training set. Moreover, 
the individual loss U( 0 ) typically tells us how close 
the predicted label /(xj) is to the true label j/,, by 
means of the signed distance to the decision bound¬ 
ary. Therefore, one may think of incorporating loss 
constraints for a certain set of users, and consequently, 
prevent individual users originally classified as positive 
(by the unconstrained classifier) from being classified 
as negative by the constrained classifier. To do so, we 
find the decision boundary parameters 9 as: 

minimize | i YliLi ( z i ~ z ) 0 Tx i I 

subject to Li(9) <{l + 'y i )L i (9*) Vi £ {1,..., N}, 

( 8 ) 

where L,(9*) is the individual loss associated to the 
*-th user in the training set provided by the uncon¬ 
strained classifier and 7 * > 0 is her allowed addi¬ 
tional loss. For example, in the case of logistic re¬ 
gression classifier, 9* = argmin e Y^iLi ~ l°gp(?/i| x i> 9) 
and the losses for individual points are Li(9) = 
— logp(?/j|xj, 9). Now, if we set 7 j = 0, we are enforc¬ 
ing that the probability of the i- th user to be mapped 
in the positive class to be equal or higher than in the 
original (unconstrained) classifier. 

4 EVALUATION 

We evaluate our framework on several synthetic and 
real-world datasets. We first experiment with our first 
formulation and show that it allows for fine-grained 
fairness control, often at a minimal loss in accuracy. 
Then, we validate our second formulation, which al¬ 
lows maximizing fairness under accuracy constraints, 


4.1 Experiments on Synthetic Data 
Fairness constraints vs accuracy constraints. To 

simulate different degrees of disparate impact in classi¬ 
fication outcomes, we generate two synthetic datasets 
with different levels of correlation between a single, bi¬ 
nary sensitive attribute and class labels. We then train 
two types of logistic regression classifiers: one type 
maximizes accuracy subject to fairness constraints 
(Section |3.2[ ), and the other maximizes fairness under 
fine-grained accuracy constraints (Section 3.3). 


Specifically, we generate 4,000 binary class labels 
uniformly at random and assign a 2 -dimensional 
user feature vector per label by drawing sam¬ 
ples from two different Gaussian distributions: 
p(x\y=l ) = AT([2; 2], [5, 1; 1, 5]) and p(x\y=-l) = 
IV([—2; — 2], [10, 1; 1, 3]). Then, we draw each user’s 
sensitive attribute z from a Bernoulli distribution: 

p(z = 1) = p( x '|y = i)/(p( x '|y = 1) + p( x 'l?/ = -1)), 

where x' = [cos(^), — sin(^); sin(<^), cos(</>)]x is sim¬ 
ply a rotated version of the feature vector, x. 


We generate datasets with two values for the parame¬ 
ter </>, which controls the correlation between the sen¬ 
sitive attribute and the class labels (and hence, the 
resulting disparate impact). Here, the closer cf> is to 
zero, the higher the correlation. Finally, we trained 
both types of constrained classifiers on each dataset. 

Fig. [Ta| shows the decision boundaries provided by the 
classifiers that maximize accuracy under fairness con¬ 
straints for two different correlation values and two 
(successively decreasing) covariance thresholds, c. We 
compare these boundaries against the unconstrained 
decision boundary (solid line). As expected, given the 
data generation process, fairness constraints map into 
a rotation of the decision boundary (dashed lines), 
which is greater as we decrease threshold value c or 
increase the correlation in the original data (from 
<j) = 7t/4 to (f> = 7t/8). This movement of the deci¬ 
sion boundaries shows that our fairness constraints are 
successfully undoing (albeit in a highly controlled set¬ 
ting) the rotations we used to induce disparate impact 
in the dataset. Moreover, a smaller covariance thresh¬ 
old (a larger rotation) leads to a more fair solution, 
although, it comes at a larger cost in accuracy. 

Fig. pT| shows the decision boundaries provided by the 
classifiers that maximize fairness under fine-grained 
accuracy constraints. Here, the fine-grained accuracy 
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Acc=0.87; p%-rule=45% 


Acc=0.87; p%-rule=24% 


Acc=0.87; p%-rule=45% 1 

Acc=0.87; p%-rule=24% 

■■■ Acc=0.82; p%-rule=70% 


■■■ Acc=0.71; p%-rule=62% 


- ■ - Acc=0.78; p%-rule=76% 

■■■ Acc=0.78; p%-rule=54% 

. Acc=0.74; p%-rule=98% 


. Acc=0.60; p%-rule=99% 


. Acc=0.64; p%-rule=93% | 

. Acc=0.53; p%-rule=99% 



(a) Maximizing accuracy under fairness constraints 




( t > = 7 t/4 (j) = 7r/8 

(b) Maximizing fairness under accuracy constraints 


Figure 1: The solid light blue lines show the decision boundaries for logistic regressors without fairness constraints. 
The dashed lines show the decision boundaries for fair logistic regressors trained (a) to maximize accuracy under 
fairness constraints and (b) to maximize fairness under fine-grained accuracy constraints, which prevents users 
with z = 1 (circles) labeled as positive by the unconstrained classifier from being moved to the negative class. 
Each column corresponds to a dataset, with different correlation value between sensitive attribute values (crosses 
vs circles) and class labels (red vs green). 


constraints ensure that the users with z = 1 classi¬ 
fied as positive by the unconstrained classifier (circles 
above the solid line) are not labeled as negative by 
the fair classifier. The decision boundaries provided 
by this formulation, in contrast to the previous one, 
are rotated and shifted versions of the unconstrained 
boundary. Such shifts enable the constrained classi¬ 
fiers to avoid negatively classifying users specified in 
the constraints. 

We also illustrate how the decision boundary of a non¬ 
linear classifier, a SVM with RBF kernel, changes un¬ 
der fairness constraints in Appendix |B.1[ 


4.2 Experiments on Real Data 
Experimental Setup. We experiment with two real- 
world datasets: The Adult income dataset Adult 


data 

1996 

data 

2014 


and the Bank marketing dataset Bank 


The Adult dataset contains a total of 
45,222 subjects, each with 14 features (e.g., age, ed¬ 
ucational level) and a binary label, which indicates 
whether a subject’s incomes is above (positive class) 
or below (negative class) 50K USD. For this dataset, 
we consider gender and race, respectively, as binary 
and non-binary (polyvalent) sensitive attributes. The 
Bank dataset contains a total of 41,188 subjects, each 
with 20 attributes (e.g., marital status) and a bi¬ 
nary label, which indicates whether the client has sub¬ 
scribed (positive class) or not (negative class) to a term 
deposit. In this case, we consider age as (binary) sensi¬ 
tive attribute, which is discretized to indicate whether 
the client’s age is between 25 and 60 years. For de¬ 
tailed statistics about the distribution of different sen¬ 
sitive attributes in positive class in these datasets, we 
refer the reader to Appendix |B.2| 


For the sake of conciseness, while presenting the results 
for binary sensitive attributes, we refer to females and 
males, respectively, as protected and non-protected 
groups in Adult data. Similarly, in Bank data, we 


refer to users between age 25 and 60 as protected and 
rest of the users as non-protected group. In our exper¬ 
iments, to obtain more reliable estimates of accuracy 
and fairness, we repeatedly split each dataset into a 
train (70%) and test (30%) set 5 times and report the 
average statistics for accuracy and fairness. 


Maximizing accuracy under fairness con¬ 
straints. First, we experiment with a single binary 
sensitive attribute, gender and age, for respectively, 
the Adult and Bank data. For each dataset, we train 
several logistic regression and SVM classifiers (denoted 
by ‘C-LR’ and ‘C-SVM’, respectively), each subject 
to fairness constraints with different values of covari¬ 
ance threshold c (Section | .3.2 [ }, and then empirically 
investigate the trade-off between accuracy and fair¬ 
ness. Fig. 2a shows the (empirical) decision bound¬ 
ary covariance against the relative loss incurred by 
the classifier. The ‘relative loss’ is normalized between 
the loss incurred by an unconstrained classifier and by 
the classifier with a covariance threshold of 0. Here, 
each pair of (covariance, loss) values is guaranteed to 
be Pareto optimal, since our problem formulation is 
convex. Additionally, Fig. [2b] investigates the corre¬ 
spondence between decision boundary covariance and 
p%-rule computed on the training set, showing that, as 
desired: i) the lower the covariance, the higher the p%- 
rule the classifiers satisfy and (ii) a 100%-rule maps to 
zero covariance. 


Then, we compare our approach to a well-known 
competing method from each of the two categories 
discussed in Section [l] preferential sampling ap¬ 
proach [Kamiran and Calders] [2010 , applied to logis¬ 
tic regression (‘PS-LR^ and SVM (‘PS-SVM’), as an 
example of data pre-processing, and the regularized lo¬ 


gistic regression (‘R-LR’) Kamishima et al. 2011 


as 


an example of modifying a classifier to limit unfairness. 
Fig.|2c|summarizes the results: the top panel shows av- 
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Empirical covariance 

Adult and Bank 


(a) Loss vs. Cov. 


c-lr - c-svm - 



C-LR - C-SVM - 



Empirical covariance 

Adult and Bank 


(b) Cov. vs. p%-rule 


C-LR C-SVM - - - ■ R-LR - PS-LR □ PS-SVM o 





Adult Bank 

(c) Single binary sensitive attribute 


Figure 2: [Maximizing accuracy under fairness constraints: single, binary sensitive attribute] Panels in (a) show 
the trade-off between the empirical covariance in Eq. [2] and the relative loss (with respect to the unconstrained 
classifier), for the Adult (top) and Bank (bottom) datasets. Here each pair of (covariance, loss) values is 
guaranteed to be Pareto optimal by construction. Panels in (b) show the correspondence between the empirical 
covariance and the p%-rule for classifiers trained under fairness constraints. Panels in (c) show the accuracy 
against p%-rule value (top) and the percentage of protected (dashed) and non-protected (solid) users in the 
positive class against the p%-rule value (bottom). 
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Figure 3: [Maximizing accuracy under fairness con¬ 
straints: non-binary and several sensitive attributes] 
The figure shows accuracy (top) and percentage of 
users in positive class (bottom) against a multiplica¬ 
tive factor a € [0,1] such that c = ac*, where c* de¬ 
notes the unconstrained classifier covariance. 


erage accuracy and the bottom panel the percentage 
of protected (dashed lines) and non-protected (solid 
lines) users in positive class against the average p%- 
rule, as computed on test sets. We observe that: i) the 
performance of our classifiers (C-LR, C-SVM) and reg¬ 
ularized logistic regression (R-LR) is comparable, ours 
are slightly better for Adult data (left column) while 
slightly worse for Bank data (right column). However, 


R-LR uses sensitive attribute values from the test set 
to make predictions, failing the disparate treatment 
test and potentially allowing for reverse discrimina¬ 
tion Ricci vs. DeStefano 2009 ; ii) the preferential 


sampling presents the worst performance and always 
achieves p%-rules under 80%; and, (iii) in the Adult 
data, all classifiers move non-protected users (males) 
to the negative class and protected users (females) to 
the positive class to achieve fairness, in contrast, in 
the Bank data, they only move non-protected (young 
and old) users originally labeled as positive to the neg¬ 
ative class since it provides a smaller accuracy loss. 
However, the latter can be problematic: from a busi¬ 
ness perspective, a bank may be interested in finding 
potential subscribers rather than losing existing cus¬ 
tomers. This last observation motivates our second 
formulation (Section 3.31, which we experiment with 
later in this section. 


Finally, we experiment with non-binary (race) and 
several (gender and race) sensitive attributes in Adult 
dataset. We do not compare with competing methods 
since they cannot handle non-binary or several sen¬ 
sitive attributes. Fig. [3] summarizes the results by 
showing the accuracy and the percentage of subjects 
sharing each sensitive attribute value classified as posi¬ 
tive against a multiplicative covariance factor a £ [0,1] 
such that c = ac* , where c* is the unconstrained clas¬ 
sifier covarianc^] (note that p%-rule is only defined for 


3 For several sensitive features, we compute the initial 
covariance c£ for each of the sensitive feature k, and then 
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(a) Acc and p%-rule (b) % in +ve class 

Figure 4: [Maximizing fairness under accuracy constraints] Panels in (a) show accuracy (solid) and p%-rule value 
(dashed) against 7 . Panels in (b) show the percentage of protected (P, dashed) and non-protected (N-P, solid) 
users in the positive class against 7 . 


a binary sensitive feature). As expected, as the value 
of c decreases, the percentage of subjects in the posi¬ 
tive class from sensitive attribute value groups become 
nearly equal [^] while the loss in accuracy is modest. 


Maximizing fairness under accuracy con¬ 
straints. Next, we demonstrate that our second for¬ 
mulation (Section 3.3) can maximize fairness while 
precisely controlling loss in accuracy. To this end, we 
first train several logistic regression classifiers (denoted 
by ‘y-LR’), which minimize the decision boundary co- 
variance subject to accuracy constraints over the en¬ 
tire dataset by solving problem 0 with increasing val¬ 
ues of 7 . Then, we train logistic regression classifiers 
(denoted by ‘Fine-y-LR’) that minimize the decision 
boundary covariance subject to fine-grained accuracy 
constraints by solving problem Q. Here, we prevent 
the non-protected users that were classified as positive 
by the unconstrained logistic regression classifier from 
being classified as negative by constraining that their 
distance from decision boundary stays positive while 
learning the fair boundary. We then increase y* = 7 
for the remaining users. In both cases, we increased 
the value of 7 until we reach a 100 %-rule during train¬ 
ing. Fig. a summarizes the results for both datasets, 
by showing (a) the average accuracy (solid curves) and 
p%-rule (dashed curves) against 7 , and (b) the per¬ 
centage of non-protected (N-P, solid curves) and pro¬ 
tected (P, dashed curves) users in the positive class 
against 7 . We observe that, as we increase 7 , the clas¬ 
sifiers that constrain the overall training loss (y-LR) 
remove non-protected users from the positive class and 
add protected users to the positive class, in contrast, 
the classifiers that prevent the non-protected users 
that were classified as positive in the unconstrained 
classifier from being classified as negative (Fine-y-LR) 
add both protected and non-protected users to the pos- 


compute the covariance threshold separately for each sen¬ 
sitive feature as ac£. 

4 The scarce representation of the race value ‘Other’ 
(only 0.8% of the data) hinders an accurate estimation of 
the decision boundary covariance and, as a result, the clas¬ 
sifier does not reach perfect fairness with respect to this 
sensitive attribute value. 


itive class. As a consequence, the latter achieves lower 
accuracy for the same p%-rule. 


5 DISCUSSION & FUTURE WORK 


In this paper, we introduced a novel measure of deci¬ 
sion boundary fairness, which enables us to ensure fair¬ 
ness with respect to one or more sensitive attributes, 
in terms of both disparate treatment and disparate im¬ 
pact. We leverage this measure to derive two comple¬ 
mentary formulations: one that maximizes accuracy 
subject to fairness constraints, and helps ensure com¬ 
pliance with a non-discrimination policy or law ( e.g., a 
given p%-rule); and another one that maximizes fair¬ 
ness subject to accuracy constraints, and ensures ful¬ 
filling certain business needs (e.g., disparate impact’s 
business necessity clause). 


Our framework opens many avenues for future work. 
For example, one could include fairness constraints 
in other supervised (e.g., regression, recommendation) 
as well as unsupervised (e.g., set selection, ranking) 
learning tasks. Further, while we note that a decreas¬ 
ing covariance threshold corresponds to an increas¬ 
ing (more fair) p%-rule, the relation between the two 
is only empirically observed. A precise mapping be¬ 
tween covariance and p%-rule is quite challenging to 
derive analytically since it depends on the specific clas¬ 
sifier and the dataset being used. Such a theoretical 
analysis would be an interesting future direction. Fi¬ 
nally, in this paper we consider disparate impact as 
our notion of fairness with the assumption that the 
historical training data may contain biases against cer¬ 
tain group(s). Since the actual proportions in positive 
class from different groups (e.g., males, females) in the 
ground-truth dataset are not available (we only have 
access to a biased dataset), ensuring equal proportions 
from each group in the positive class (removing dis¬ 
parate impact) serves as an attractive notion of fair¬ 
ness. However, in cases where the historical ground 
truth decisions are available, disparate impact can be 
explained by the means of the ground truth, and al¬ 
ternative notions of fairness, e.g., disparate mistreat¬ 
ment Zafar et al. 2017 , might be more suitable. 
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Fairness Constraints: Mechanisms for Fair Classification 


A Particularizing the Fairness Constraints for SVM 


One can specialize our fair classifier formulation proposed in Q as: 

Linear SVM. A linear SVM distinguishes among classes using a linear hyperplane 0 T x = 0. In this case, the 
parameter vector 6 of the fair linear SVM can be found by solving the following quadratic program: 


minimize ||0|| 2 + C £)" =1 & 
subject to y.iO T Xi > 1 — G {1,... ,n} 
& > 0, V* G {1,..., n}, 

w E^=i ( z i - z ) < c 
iEh (Zi - z) 0 T x l > -c, 


(9) 


where 6 and £ are the variables, ||0|| 2 corresponds to the margin between the support vectors assigned to different 
classes, and C E”— 1 £,i penalizes the number of data points falling inside the margin. 


Nonlinear SVM. In a nonlinear SVM, the decision boundary takes the form 0 T $(x) = 0, where $(•) is a 
nonlinear transformation that maps every feature vector x into a higher dimensional transformed feature space. 
Similarly as in the case of a linear SVM, one may think of finding the parameter vector G by solving a constrained 
quadratic program similar to the one defined by Eq. <©• However, the dimensionality of the transformed feature 
space can be large, or even infinite, making the corresponding optimization problem difficult to solve. Fortunately, 
we can leverage the kernel trick Scholkopf and Smola 2002 both in the original optimization problem and the 
fairness inequalities, and resort instead to the dual form of the problem, which can be solved efficiently. In 
particular, the dual form is given by: 


minimize Eill a i + Eili ai2/i(5a( x i) + /i«( x i)) 

subject to OLi > 0, Vt G {1,..., V}, 

Eill a iVi = 0> 


lEf=l( Z i- Z )?a( X .) <C, nn x 

1 N , — \ / \ ^ l iu J 

^Ei=l (Zi-z)Sa(x*) > -C, 

where a. are the dual variables, g a (xi) = Eyli x j) can still be interpreted as a signed distance to the 

decision boundary in the transformed feature space, and /i„(xj) = Eyli a jVj w here Sij = 1 if i = j and 
dij = 0, otherwise. Here, fc(xj,Xj) = (0(xj), <p(x.j)) denotes the inner product between a pair of transformed 
feature vectors and is often called the kernel function. 


B Additional Experiments 

B.l Experiments on Non-linear Synthetic Data 

Here, we illustrate how the decision boundary of an non-linear classifier, a SVM with radial basis function (RBF) 
kernel, changes under fairness constraints. To this end, we generate 4,000 user binary class labels uniformly at 
random and assign a 2-dimensional user feature vector per label by drawing samples from p(x\y = l,/3) = 
PN([ 2; 2], [5 1; 1 5]) + (1 - p)N([- 2; -2], [10 1; 1 3]) if y = 1, and p(x\y = -1, /3) = 0N([ 4; -4], [4 4; 2 5]) + (1 - 
/3)At([—4;6], [6 2; 2 3]) otherwise, where /3 G {0,1} is sampled from Bernoulli^ 0.5). Then, we generate each 
user’s sensitive attribute 2 by applying the same rotation as detailed in Section |4~H 

Figure [5] shows the decision boundaries provided by the SVM that maximizes accuracy under fairness constraints 
with c = 0 for two different correlation values, set by <f> = it/ 4 and <j> = 7r/8, in comparison with the unconstrained 
SVM. We observe that, in this case, the decision boundaries provided by the constrained SVMs are very different 
to the decision boundary provided by the unconstrained SVM, not simple shifts or rotations of the latter, and 
successfully reverse engineer the mechanism we used to generate the class labels and sensitive attributes. 

B.2 Experiments on Real Data 

Additional Data Statistics. In this section, we show the distribution of sensitive features and class labels in 
our real-world datasets. 
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Figure 5: Decision boundaries for SVM classifier with RBF Kernel trained without fairness constraints (left) 
and with fairness constraints (c = 0 ) on two synthetic datasets with different correlation value between sensitive 
attribute values (crosses vs circles) and class labels (red vs green). 


Table 2: Datasets details (binary sensitive attributes: gender and age). 


Sensitive Attribute 

y < 50 K 

> 50 K 

Total 

Males 

20,988 

9,539 

30,527 

Females 

13,026 

1,669 

14,695 

Total 

34,014 

11,208 

45,222 


(a) Adult dataset 


Sensitive Attribute 

No 

Yes 

Total 

25 < age < 60 
age < 25 or age > 60 

35,240 

1,308 

3,970 

670 

39,210 

1,978 

Total 

36,548 

4,640 

41,188 


(b) Bank dataset 
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(a) Acc and CV score (b) % in +ve class 

Figure 7: [Maximizing fairness under accuracy constraints] Panels in (a) show the accuracy (solid) and CV score 
value (dashed) against 7 . Panels in (b) show the percentage of protected (P, dashed) and non-protected (N-P, 
solid) users in the positive class against 7 . 


CV score as fairness measure. While evaluating the performance of our method in Section]!) we used p%-rule 
as the true measure of fairness, since it is a generalization of the 80%-rule advocated by US Equal Employment 
Opportunity Commission [Biddle| 2005] to quantify disparate impact. We would like to remark that this measure 
is closely related to another measure of fairness used by some of the previous works [Kamiran and Calders 2009 


Kamishima et al. 2011 |Zemel et al. 2013 in this area, referred to as Calder-Verwer (CV) score by Kamishima 


et al. 2011 . In particular, the CV score is defined as the (absolute value of the) difference between the percentage 


of users sharing a particular sensitive attribute value that lie on one side of the decision boundary and the 
percentage of users not sharing that value lying on the same side, i.e., | P(dg(x) > 0\z = 0 ) — P(dg(x) > 0\z = 1)|. 

In this section, we show that using CV score (instead of p%-rule) as a measure of fairness would yield similar 
results. 


First, we show that constraining the covariance between users’ sensitive attributes (Fig. 6 a and Figure [ 6 b| , and 
the signed distance from the decision boundary, corresponds to an increasing relative loss and decreasing CV 
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Table 3: Adult dataset (Non-binary sensitive attribute: race) 


Sensitive Attribute 

y < 50A' 

> 50 K 

Total 

American-Indian/Eskimo 
Asian/Pacific-Islander 
White 

Black 

Other 

382 

934 

28,696 

3,694 

308 

53 

369 

10,207 

534 

45 

435 
1,303 
38, 903 
4,228 
353 

Total 

34,014 

11,208 

45,222 
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(c) Single binary sensitive attribute 


Figure 6: [Maximizing accuracy under fairness constraints: single, binary sensitive attribute] Panels in (a) show 
the trade-off between the empirical covariance and the relative loss in accuracy (with respect to the unconstrained 
classifier), where each pair of (covariance, loss) values is guaranteed to be Pareto optimal by construction. Panels 
in (b) show the correspondence between the empirical covariance in Eq. [2] and the CV score for classifiers trained 
under fairness constraints for the Adult (top) and Bank (bottom) datasets. Panels in (c) show the accuracy 
against CV score value (top) and the percentage of protected (dashed) and non-protected (solid) users in the 
positive class against the CV score value (bottom). 


score (a more fair decision boundary). 

Next, we show the performance of different methods based on the CV score (Fig,[6c|and[7|). The results in Fig 6c 
and [7] correspond to the ones shown in Fig. [2c|and|4} where we took p%-rule as the measure of fairness. It can 
be seen that both measures of fairness (p%-rule and CV score) provide very similar trades-off in terms of fairness 
and accuracy. 


Notice that according to the definitions provided in Section |2.1[ a decreasing CV score corresponds to an in¬ 
creasing p%-rule (and hence, a more fair decision boundary). 






































